In this lesson we're going to talk about simplifying our model. 1
And that's because, all else equal, simpler models are preferable to complex ones. 2
Remember the Zen of Python? "Simple is better than complex." and "Complex is better than complicated." and 3
this doesn't just apply to programming. 4
The same thing can be said about our regression model. 5
By the way, you can always bring up this little gem of programming philosophy when you type "import this" 6
in Jupyter notebook. Simple models really are preferable to complex ones, 7
all else equal. 8
So the question is: how can we simplify our model? 9
One of the easiest ways is to remove some of the explanatory variables, but can we just drop some features? 10
Is that a wise thing to do? 11
And if so, which features should we drop? 12
What about the features that were not highly correlated with property prices? In the correlation matrix, 13
we saw that distance from employment centers only had a 0.25 correlation with our target. 14
DIS had a low correlation with price but also it had a high correlation with our industry factor of 15
-0.71. 16
At the time we were wondering how much value that distance from employment centers feature really added. 17
But now we know. Scrolling down, 18
we've got the p-values which test for the significance of our different factors and we can see that 19
distance is actually very statistically significant. 20
So we should probably keep DIS around. Now on the other hand, looking back up at our industry factor, 21
this has a p-value of around 0.44 meaning it is not statistically significant. 22
The threshold for p-values, if you recall, was 0.05. 23
So now the question is: should we try removing the industry factor from the model? 24
And the thing is, it is really really tempting to remove insignificant predictors. 25
But even dropping statistically insignificant features is not something people do lightly, because even 26
a feature with a low p-value can add value to the model as a whole by providing some kind of information 27
that the other features do not provide. 28
Actually deciding on what to keep and what to throw away from a machine learning model is a bit of an 29
art and it gets into this whole topic of feature selection which is a very, very big topic indeed and 30
one that we will continue to tackle throughout this course. 31
The goal of this lesson is to introduce you to this topic of feature selection in the context of a regression 32
model. And we will be looking at a metric that you can use to help you make your decisions. 33
And that metric is called Baysian information Criterion or BIC. The Baysian information criterion 34
is a way you can measure complexity. 35
It's basically a number that allows you to compare two different models. 36
So what you end up doing is you run a regression with model number one and then you run a regression 37
with model number two. Now model number one might have a BIC value of 148 and model number two might 38
have a big value of 154. The actual number doesn't by itself mean very much. 39
What matters is which one is lower, because all else equal a lower BIC number is better. So this measure 40
can help you pick between two or more models. 41
So what models will we compare? Well for starters, let's compare the model that includes the industry 42
feature and the model that excludes the industry feature. 43
Let's commemorate this with another markup cell in Jupyter notebook. So I'm going to change my cell here to Markdown 44
and then put a section heading here 45
"Model Simplification & the Baysian Information Criterion". To calculate the Baysian information criterion, 46
we're once again going to use our statsmodel module's regression capabilities instead of scikit-learn 47
so let's copy this cell here and paste it below. 48
I'm going to delete these comments and I'm going to add a new comment up here 49
that reads "Original model with log prices and all features" and the data frame that I've got here 50
I'm going to store inside a variable. So I'm going to say "original_coef = pd.DataFrame()" 51
and so on. And now what I want to do is I want to add some additional print statements and in these 52
print statements we'll output both the Baysian information criterion value and the r-squared for the 53
regression. 54
Now previously we've used scikit's learn score method to print out the r-squared but things work a little 55
differently with the statsmodel module and I think this is a good time to practice making sense of 56
the official documentation. 57
So as a challenge, can you look up the statsmodel docs for the regression results and figure out how 58
to print out both the Baysian information criterion value for this regression as well as the r-squared? 59
I'll give you a few seconds to pause the video and give this a shot. 60
You ready? 61
Here's the solution. 62
Being able to read and interpret the official documentation for all a lot of these Python modules is 63
one of the key skills at becoming a better programmer. 64
If I click on my results object and press Shift+Tab on my keyboard to bring up the quick documentation, 65
I don't actually get anything useful. 66
I just see that to get further information I need to look at the regression results documentation. 67
The same thing is true if I click here and I find out that I'm just dealing with a dataframe or when 68
I click here. 69
So in all these cases, the quick documentation isn't actually helping me all that much. I'm just not having 70
any luck on getting the relevant information. 71
So what I'm going to have to do is I'm going to have to Google the documentation myself. 72
The best keywords to enter into that white text box are "statsmodel regression results" and that should 73
pretty much take you to one of the statsmodel pages. 74
Now out of these three that I've got here, the one I'm looking for is the page for the documentation 75
on the RegressionResults object. 76
So it's the third one down here. 77
This is the regression results object from the linear model. 78
This is the documentation page that you want to be looking for. 79
The other reason why the RegressionResults page that I've clicked on is the relevant one is because 80
this particular object, the RegressionResultsWrapper, inherits most of the methods and attributes from 81
RegressionResults, meaning the capabilities of a RegressionResultsWrapper object which we've got and 82
a RegressionResults object are pretty much the same. 83
They have a lot in common, a lot of the methods and attributes are inherited from this particular object. 84
Yeah they're closely linked. 85
So this is why I'm looking on this page. Now, 86
this is an incredibly long page if you look at it, it's very, very comprehensive, but the interesting thing 87
is that we can find both the params attribute and p-values attribute on this page. 88
So if I search for params on this page, I can find params listed as one of the attributes and 89
I can also find p-values listed as one of the attributes. And you probably already spotted it at the 90
top of the page is the BIC attribute, the Bayes information criteria, so BIC is the name of the 91
attribute for the Bayes information criteria and the way we access this attribute is simply by writing 92
"results. 93
bic" 94
and I if I hit Shift+Enter, I can see what the value of this actually is - it's 95
-139.85 so about -140. And what about the r-squared? 96
Going back to the documentation and scrolling down, r-squared unsurprisingly has the attribute name rsquared, 97
all lowercase and in one word. So "results.rsquared" will bring up the r-squared for this regression 98
which is 0.793. 99
Let's have both of these lines of code in a print statement. 100
So I'm going to have "print('BIC is', results.bic)" and then 101
I'll have "print( 102
'r-squared is', results.rsquared)". 103
There we go. 104
Okay, so now we have both the r-squared and our BIC printed out. The comforting thing to see is that the 105
statsmodel and scikit-learn is exactly the same r-squared. 106
So we're doing things right. Now in this case, the Baysian information criterion is actually a negative 107
number and that's absolutely fine. 108
What matters is how this number stacks up to our next model. 109
So I'm going to copy this, paste it and then I'm going to modify my comment, 110
I'm going to write here "Reduced model #1 excluding INDUS" and then what I'll say is that 111
"X_incl_const = X_incl_const.drop( 112
['INDUS'], axis=1)". 113
So in this line of code I'm redefining the dataframe of features by overwriting what's stored inside 114
this variable. 115
I'm dropping the INDUS column from the dataframe and storing that as the new feature dataframe. 116
So on this line when it comes to training our model we are excluding the INDUS feature. Next, I'm going 117
to change the name of this variable to "coef_minus_indus", 118
so that way we're not overwriting the coefficient data frame from the cell above 119
and I'm going to delete this comment here. 120
Now let me hit Shift+Enter and refresh this cell. 121
So this result is already quite interesting. 122
What we can see is that our Baysian information criterion has gotten more negative, 123
we've now got the value minus -145.2, 124
so this is an even lower number than before. 125
So we have an improvement in terms of reducing complexity, but at the same time it's also really nice 126
to see that the r-squared at 0.79 127
pretty much stays where it is. 128
So even though we have removed one feature from our dataset, it hasn't really impacted our fit in a material 129
way. 130
This is actually very encouraging. 131
Let's go back up to our p-values and experiment with removing something else. 132
Let's experiment with removing AGE. Coming back down, I'm going to copy this cell, paste it, change my 133
comment here to "Reduced model #2 excluding INDUS and AGE" and then in this line I'm going to 134
have to add the single quotes and AGE between the square brackets. 135
I'm also going to rename our dataframe of coefficients to maybe "reduced_coef" and now I'm going to hit Shift+ 136
Enter and what we actually see is a further improvement based on the Baysian information criterion. 137
So we get an even lower BIC number at -149.5 , but we see no material change in 138
the r-squared. 139
So this makes me think that removing both INDUS and AGE is actually a beneficial thing. 140
We can probably safely drop these two features simplifying our model without incurring too much of a 141
cost in terms of lost information and a worse fit. 142
Now even though I just gave you two examples where removing a feature improved the Baysian information 143
criterion and left the r-squared pretty much unchanged, 144
this isn't always the case. If I change Age to one of the other features, say maybe zone, ZN, and press 145
Shift+Enter, what we see is not really all that clear cut. 146
In this case, we have a higher big number and a lower r-squared than before. 147
So this is probably not the direction we want to go in. 148
Same thing if I change this to our TAX feature. 149
Again we're making our model worse. 150
And same thing if I change this to the LSTAT feature. Removing LSTAT actually makes our model 151
much, much worse. 152
You can see how much the Baysian information criterion jumped and how much lower our r-squared is 153
in this case. So LSTAT is actually very important to keep in the model. I'm going to change this back to AGE and 154
press Shift+Enter so that we're back to where we started. 155
Okay. 156
So where are we now? 157
We've made two small tweaks to our model. 158
We've removed two of the features which were not statistically significant and we've looked at the Baysian 159
information criterion and the r-squared to provide additional justification for leaving them out and 160
simplifying our model and by doing so we've managed to improve our BIC number from around -140 161
to -150. 162
So we get about a 9-10 point improvement in the BIC number, but we don't incur a material penalty on 163
the r-squared. 164
We're still on 0.79. 165
Cool. 166
So that about wraps up our introduction to thinking about feature selection and one thing that we can 167
do now is we can link this lesson to our previous one. 168
We can link this to the discussion on multcollinearity and looking for stability in the theta estimates 169
for our features, because we've made quite a few tweaks to our model and we said that one of the symptoms 170
of multicollinearity are unstable coefficients. 171
Having run three different versions of our regression and having stored our coefficients in some variables, 172
we can now look at them side by side and double check if there are any strange developments. 173
Now I'd be surprised if we saw any 174
because we have no indication of multicollinearity so far. 175
But take a look at this. 176
If I create a variable called "frames" and make that equal to a list of our dataframes. 177
So we had the org_coef, we had coef_minus_indus dataframe, 178
and then finally we had the reduced_coef dataframe of coefficients. To put them all side 179
by side, 180
I'm going to use pandas concat method so "pd.concat() and then in the parentheses to provide my list 181
of dataframes and an axis. So I want this to be concatenated along the columns, 182
so side by side as opposed to top to bottom, right? So axis is gonna be equal to 1 instead of 0. 183
Let's see what we get. 184
Fantastic. 185
Now what you can also see here in this table here is how Python treats missing values in a dataframe. 186
You see these nans, NaN? NaN stands for "Not a Number" meaning there is a missing value here. 187
So this column here is our reduced model without the AGE and without the Industry feature. 188
So we've got NaNs in place of these rows. 189
Looking at this table is actually very, very encouraging because what I'm seeing is that despite tweaking 190
the model all the coefficients, 191
yeah for all three, 192
so like Charles River, Crime, they all are remarkably consistent, so the numbers change somewhat but they 193
don't switch signs, 194
they don't change drastically, 195
so these are all very, very stable coefficient estimates and the same actually holds true 196
if you were to remove TAX which we suspected was a potential problem source. 197
So even removing TAX and rerunning this, you'd see that the theta estimates are nice and stable between 198
the three models. And that brings us to the end of this lesson. In the next one, 199
we're gonna take our evaluation of our regression even further. 200
We're going to look at how far off our models' predictions were from their true values. 201
We're gonna be looking and analyzing our regression residuals. Now, 202
as an aside, while putting this lesson together for you I found myself googling the word BIC and trying 203
to read up on the information criterion and for a split second, I was quite confused 204
why all I got on the front page was information about pens hand the website of the Bournemouth International 205
Centre. 206
The other time that happened to me was when I read up on nested tables then I was confronted with this. 207
So yeah if you have any stories like this to share, please put them in the comments section for this 208
video. 209
I'd love to hear it. See you in the next lesson.